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ABSTRACT 

Recognizing the demand for teaching concepts in 
teacher education programs^ the Bureau of Educational Personnel 
Development in 1970 initiated an effort to train educational 
personnel to develop and use protocol materials-^ — instructional 
materials y usually employing audio tape, video tape, or film, 
intended to illuminate a concept by showing instances to which the 
concept correctly applies. These instances typically involve the 
behaviors of children and adults as they appear in the classroom or 
in other community settings in which teachers might be expected to 
interpret behavior for purposes of education. - Description is given of 
this protocol materials effort which is essentially a training 
program for project directors. The materials being produced^ 
primarily films^ cover the following subject matter areas: 
educational psychology^ reading^ literature^ language acquisition^ 
Black English^ social psychology^ teaching analysis^ and social 
studies. , Appendices list the eleven pilot projects which were funded 
initially and described a progressive evaluation system^ developed by 
Richard L. Turner^ which currently is used in training project 
directors as they develop protocol materials. . (Author/SHM) 
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The impetus for the kind of educational change that truly iu^roves 
the classroom setting must either be initiated by or closely involve 
teachers. All too often theories concerned with educational change 
have resulted in innovation which exists seemingly only frjr its own 
sake. Frequently the educational theorist is the university 
professor, who, long away from the classroom, suggests r^ange based 
upon the soundest of theory and the most iiopractical of liiethod. In 
the classroom the effort leads at best to more theorizing and at worst 
to disaster. 

seettd apparent that worthwhile and lasting change can 
occur only through the joint efforts of university staffs, 
administrators, supervisors, and classroom teachers. Preparing 
individuals for the dual role of competent teaching and educational 
planning is the responsibility of the teacher education program, 
for the final test of the validity of educational planning is what 
happens in the classroom. That teachers acquire knowledge and skill 
on the job is patently clear; however, they should enter the 
classroom already equipped with certain understandings and skills. 

Universities and school systv-^ms , the institutions that train 
educational personnel recognise the need for materials that will 
help supervisors and teachers acquire these understandings and 



2. 



skills. Although innumerable written materials exist in education 
and allied fields , few appear to induce desired behaviors in pre- 
or in-service teachers. 

Traditionfilly , the teacher *s primary function at any level 
has been to help students learn. If teachers are to help students 
learn, however, they Toust be prepared for the task of teaching; 
they must know the subject matter, and they must know how to teach. 
Additionally, the teacher must not only understand the students but 
he also must understand the learning process. Individuals differ 
in learning rates and respond to teachxng-learning situations in 
various ways. 

Developing understandings about learning is not an ^A§y task. 
Extensive observation of students is essential, yet observation 
without focus accomplishes little. The observations must be 
directed; the teacher must look for particular kinds of behaviors. 
There Is no guarantee that a given stimulus will result in the 
behavior the teacher wishes to observe; he may observe a class for 
an entire school day and finu no evidence of a desired behavior. 
Even if a particular behavior is exhibited, the instance may be 
fleeting and the observer luw only his recollection for considered 
study. 

ERLC 
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Preparation for interpreting behavior usually consists of 
courses presented in the traditional reading-lecture-discussion 
manner which often fail to achieve their goals because the in- 
struction is divorced from reality, Concep:;s and principles fre^ 
quently are taught in abstraction with little opportunity 
provided the teacher for applying them in meaningful situations. 
One liieans of bridging the gap between theory and reality is reproduc- 
ing a variety of behaviors of students, teachers, and others in a 
permanent recallable form, A particular segment of behavior can 
be reproduced again and again to be studied, analyzed, and the con- 
cepts appropriate to its interprstation spotlighted, explained, 
learned, and reviewed. The concepts can then be applied to the 
understanding of other behaviors. Instructional materials of this 
type are referred to as protocol materials. 

Protocol materials constitute one of two general categories 
of instructional materials for teachers — one: the materials that 
direct the pre- or in-service teacher in studying his and others' 
behavior, and two: those materials that guide him in systematic 
practice of the skills he must acquire. Materials that enable the 
teacher to study behavior are referred to as protocol materials. 
Training materials, on the other hand, are designed to help the 
teacher in the acquisition of skills and provide for (1) identifi- 
cation of skills, (2) description of behavior ei\t4ii^^ fehfe 
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skills, (3) performance of the behavior, and C4) feedback to the 
performer and further performance by him. 

Cognitions, however, are developed primarily through th.e 
use of protocol uiaterials which provide for (1) segments of 
behavior categorized for the purpose of teaching concepts and 
principles used in interpreting behavior as well as the social con- 
text in which the teacher works, (2) segments of behavior ca^~;orized 
for the purpose of teaching knowledge about knowledge, ^ (3) segments 
of behavior categorized for the purpose of teaching ? -understanding. 
According to Smith, the study of protocol materi Is not only results 
in the teacher ability to understand or interpret situatioAs 
faces in the classroom, school, and community, but it also increases 

5 

his interest in theory as he discovers that it is useful in teaching. 

Hudgins considers a protocol the portrayal of a concept which 

is pertinent to the subject matter with which the teacher deals or 

a concept about teaching itself. He notes that the portrayal of a 

concept is not merely a dictionary definition of the label given to 

the concept; rather, it involves laying out, through a series of 

episodes^ the characteristics that exemplify the concept to be 
A 

portrayed. 
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Protocol materials consist of reproductions of behavior — visual 
audio-visual, or printed media — that exemplify concepts relevant 
to teaching and learning. The use of protocol materials may 
enable teachers to acquire concepts that have not been learned 
particularly well when presented in the traditional manner — through 
lectures, discussion, or readings. Although the occasional 
brilliant teacher may present concepts so skillfully that they 
are acquired by prospective or ptacticing teachers, more often the 
result is exposure to concepts with no assurance that concept 
acquisition has occurred. 

Gllessman says that if teachers are to develop the ability to use 

concepts interpretively , verbal instruction alone Is almost 

certainly insufficient. Teachers should have an opportunity to 

observe and interpret on-going behavior using concepts In a 

systematic way; protocol materials provide behavior that can be 
3 

interpreted » According to Smith, if a teacher does not understand 

the nature of concepts, causes, and values, he does not know the 

subject matter of Instruction and consequently lacks the logical, 

psychological, and linguistic sophistication that will enable 

5 

him to manipulate content to the advantage of the pupil. 
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Protocol materials should bridge the gap between theory and LLie 
teaching-learning situation. Since they provide reproductions of 
behavior, they foster the teacher's Interpretive and diagnostic 
competency. Gee and Berliner say that protocol materials serve a 
dual role: first, they teach concepts, and second, they teach 
teachers how to Interpret and diagnose human behavior in terms of 
those concepts. They further divide the diagnostic component of 
protocol materials into three ^t^gflta* ^ functional knowledge of 

some psychological, philosophical, sociological,, etc., concepts 
that are relevant to the teacher's work; (2) the ability to interpret 
behavioral situations in terms of these concepts, and (3) the ability 
to use such interpretations to formulate alternative plans for 
teaching and other pertinent activities in which the teacher is 
engaged. The study of protocol materials ehotild aid teachers in 
mastering concepts which can serve as the basis for interpretation 
of behavior and for decision making. The prospective or practicing 
teacher can become familiar with a variety of instances of the . 
concepts to be taught. The materials provide an opportunity for the 
teacher to understand the defining properties of the concept and to 

identify, at an indicated level of competence, instances of the 

2 

concept that are relevant to the educational setting* 

A concept may lead directly into a skill, or it may bear no 
relationship to a skill. The concept of diagnosis, once acquired by 
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teachers, can, with appropriate training, lead to the skill of 
diagnosis — a skill which a teacher uses daily in the classroom 
in determining each student's progress in learning. Conversely, 
concepts such as r^jspect or racism do not lend themselves to parallel 
skills . 

A basic problem is precision in defining_a_concept ; a concept must 
be defined precisely and its attributes specified in terms which 
permit no misunderstanding. Carroll notes that meaning and concept 
generally have been treated as separate things by different disci- 
plines. Meaning has been considered to belong to semantics while 
"concept is alinost anybody's oyster." Concepts, according to 
Carroll, are properties of organismic experience, the "abstracted 
and often cognitlvely structured classes of mental experience 
learned by organisms in the course of their life histories." 
There are necessary conditions for the formation of a concept: the 
Individual must have a series of experiences that are similar in one 
or more respects; th6 constellation of respects in which they are 
similar constitutes the underlying concept. Positive instances 
of the concept are experiences embodying the concept, and 
negative instances are those which do not. Another essential 
condition for concept formation is that the series of experiences 
embodying the concept must be preceded, interspersed, or followed 
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by other experiences that constitute negative instances of the 
concept. As the complexity of the concepts increases, the necessity 
for an appropriate sequer:.:tng of positive and negative instances 
to assure adequate learning of the concept becomes greater. 

Concepts become more complex during the course of an Individual' 

life. Because each individual's experiences are unique to him, he 

will classify these experiences in particular- ways . Consequently, 

the critical attributes that differentiate experiences can be 

specified. The individual does not necessarily have to specify the 

attributes; for example, children may not be able to verbalize 

the similarities in experiences, but consistent responses to 

particular stimuli indicate that concept information has occurred. 

Carroll mentions the classic instance where the ehild is afraid 

of the barber because he wields instruments, acissors, that look 

like those of the doctor \^om he has already learned to fear, and 

1 

because he wears a similar white smock. 

Recognizing the demand for materials for teaching concepts in 
teacher education programs, the Bureau of Educational P^^rsonnel 
Development in 1970 initiated an effort to train educational 
personnel to develop and use protocol me-erials. Materials 
developed for training teachers generally have been prepared in 
isolation with no field testing during the development stages. 
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Since protocol materials should be attuned to problems teachers 
encounter, the Office of Education insisted that people should 
be trained to develop protocol materials with field testing 
and modification based on the field testing an integral part of the 
training program. Eleven pilot projects were funded initially, 
currently 16 projects are being supported. 

The protocol materials effort ts es3:en tidily a trainiTig piogram. 
Project directors are trained, under the aegis of the Leadership 
Training Institute on Protocol and Training Materials, to develop 
and use protocol materials. The Leadership Training Institute is 
composed of a group of consultants outside the Office of Education 
headed by B. Othanel Smith of the University of South Florida 
who are responsible for providing technical assistance to the 
project directors. The training involves several stages. The 
concepts to be exemplified in protocol materials must be selected 
and analyzed. The concepts must be critical to teacher education; 
that Is, they must be concepts that teachers need to know. A 
paramount consideration is utility; the concepts to be portrayed 
in the materials must contain an element of universality so that 
the materials will be useful at any institution or agency which 



* The eleven pilot projects are listed In Appendix A. 
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trains or retrain teachers. A developer can s\ibstantiatf; 
empirically that the concepts he chooses to exemplify meet selection 
criteria. If the final products are to be films or tapes, a 
director must learn to work with producers. 

The materials being produced, primarily films , cover the following 
subject matter areas: educational psychology, reading, literature, 
l^sJiguage acquisition. Black English, social psychology, teaching 
analysis, and social studies. 

However, the products of these efforts, important as they may 
be ;in themselves, are not the part of this OE effort. The 

important result to the future of teache:: training lies in the fact 
that these leaders in the field of tea^jher education will themselves 
have acquired^, skill in the development of protocol materials. 

A major contribution to the ^^S€60^ materials program is a 
progressive evaluation system developed by Richard L. Turner* which 
currently is used in training project directors as they develop 
protocol materials . It should become a highly useful tool in the 
future development of materials for teacher education. 

*The Progressive Evaluation of Protocol Materials Development 
appears in Appendix B. 
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The system is based on five frames Incorporating nine evaluation 
decision points. The project director answers criterial questions, 
providing evidence for the questions at each decision point. If 
sufficient evidence for each question cannot be given by the project 
director, or, in some cases, the producer or evaluator, the director 
muat return to the first frame and select another concept which 
can be carried through the progressive evaluation. 

The first frame concerns the concept selected for exemplification. 
The concept must be named, and the criteria for the concept listed 
as well as an instance of the applicatior* of the concept in a given 
situation. The dic^t^r should be able to sketch three situations 
in which the concept could be observed In^ecmtotu^^B behavior* 
If the final product is a film or video tape^thc producer also 
suggests three situations in- which the concept could be observed. 
If the concept lacks clarity and cannot be represented, the director 
either reconsiders the evidence or he returns to the beginning of 
Frame One. If the evidence provided by both the project director 
and producer indicates that the concept can be represented. Evaluation 
Decision #1 is reached, and the director proceeds to Frame Two, 
the significance of the concept. 
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The significance of the concept is probably the most crucial 
issue in the development of protocol materials. A danger lies in 
working with private concepts, i*e., meaningful only to the director. 
If the concept appears in three textbooks in the field, it is not 
a private concept but possesses a degree of universality. The 
importance of the concept to teaching can be substantiated by citing 
three research studies or scholarly papers in which the concept was 
investigated. A further indication that the concept is not private 
is its theoretical relationship to other concepts. Since the concept 
is part of a theoretical framework, the director must be able to 
list the main concepts to which his concept is theoretically 
related. If there is substantial evidence that the concept is 
significant. Evaluation Decision //2 is reached, and the director 
moves dn to Frame Three. If the concept is peripheral, the director 
either must reconsider the evidence or return to the start* 
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Frame Three is concerned with learning objectives. The 
.learning outcomes desired as a consequence of using the protocol 
must be specified. If the outcome is limited to concept acquisition, 
the director and/or evaluator list the specifications for testing 
it. The target population must be indicated. This population may 
include both pre- and in-service teachers of English or history; 
it may be pre-service teachers in an educational psychology program, 
or it may include in-service teachers of mathematics. A target 
population of '^pre- or in-service teachers" without qualifiers 
is too broad. A test format should be prepared? which, in its 
format assures the presence of the cross-validity film or tape . 
materials and a practical method 'of obtaining student responses. 
This test can be used to assess the degree of concept acquisition. 
The target level of performance should be specified; the director's 
goal may be 100% identification of the concept or only 80%. The 
required performance context should also be specified; the 
films may be self-contained or they may be accompanied by printed 
materials* 

If the outcome is to be concept generalization and mastery, 
the evidence for the questions for concept acquisition listed above 
must be provided. In addition mastery must be defined in terms 
of the scope of the project. The scope of the cross validity 
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materials should give a wide range of instances in which the concept 
is to be recognized. If the test of concept mastery is to be 
a segment of film, plans for sufficient footage must be incor- 
porated from the beginning of the project. The director must 
indicate the context and the type of performance required by the 
users to assure that concept mastery has occurred. At Evaluation 
Decision point thret; the developer may decide that he needs help 
in clarifying his learning objectives, they have been achieved, 

the director proceeds to Frame Four which is concerned with the 
anticipated user. 
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18. 

Generally, the director has a group of users, instructors of pre- 
or in-service courses, in mind. However, generalizing that the materials 
may be used by either prospective or practicing teachers or both is an 
insufficient basis for assuming that the materials will be useful. The 
director should list three specific courses (or modules if a modular 
program is used) in which his protocol materials will be significant. 
Since his materials probably will not constitute an entire course, he 
should determine the proportion of a course that T^lli focus on the 
concept exemplified in the materials. His judgment of the time re- 
quired can be verified by course instructors in determining what 
the learning, outcomes are worth in terms of time. If the evidence 
indicates that the director consulting with the producers can accomplish 
his learning objectives in that specified period of time, he has com- 
pleted Evaluation Decision #4 and moves on to Decision //5 . However, 
if he cannot produce such evidence, he may have to redesif.n liis package t( 
better incorporate principles of concept learning. The director 
may also need to discuss with a production expert as well as the 
producer the possibilities of improved production techniques. If 
the director finds that he is unable to meet the user requirements, he 
returns to Frame One; if he meets them, he moves into the produc- 
tion phase. 
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22. 

Frame five is concerned with the evaluation of the results of 
using the protocol materials. The first criterial question deals 
with the director's preparation of multiple equivalent forms of the 
main stimulus materials. Three types of tests should be available. 
The first is an entry test based on the stimulus materials which 
can be used to place students in terms of their knowledge of the 
concept; if, at this stage, some students exhibit mastery of the 
concept, they need proceed no further. The second is a test of^ 
concept acquisition composed of material different from the learning 
materials. If the learning objectives require concept mastery, 
the third test, which is composed of complex stimuli from which the 
concept indicators must be discriminated, is used. Turner con- 
siders the difference between concept acquisition and concept 
mastery as one of degree. 

Exact instructions for users of the protocol materials must 
be prepared so that the protocol materials package will be used 
properly. A 'kit" or manual telling the user how to use the 
protocols package must include directions for administering the 
tests in order to obtain reliable results. If the production 
design does not yield the appropriate materials, the director may 
decide to re-edit or to produce more material. When the materials 
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for evaluating results of the protocols are completed. Evaluation 
Decision #6, the director moves on, 

The next criterial question concei'ns identification of a 
sample of users for the field test; the sample must include 
more than the director's class. The director and evaluator should 
provide a list of at least three potential users for the field 
trial. Evidence for the next two criterial questions is provided 
by the evaluator. Students of the users in the target population should 
be specified under the learning objectives; the major characteris- 
tics of the students should be listed. To provide evidence for the 
criterial question, has the user an appropriate required performance 
context, the evaluator describes this context. 

This step brings the director to Evaluation Decision #7; if 
the field situation is not appropriate, another group of users must 
be found for the field trial. If both the users and context are 
congruent with the learning objectives, the director moves on 
to the next evaluation decision point where the evidence again is 
supplied by the evaluator. He determines if the students of each 
user can be divided into a treatment group (groups) and a control 
group and determines the method by which the treatment and control 
subjects were assigned. Evaluation Decision #8 is now 
reached, and if the treatment and control sufej^cta were randomly 
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assigned, the director moves ahead. If the ass,ignment was not random, 
the evaluator must decide whether or not to employ a weaker design 
such as the intact groups, or a single group design with no controls. 
The group or groups are tested either for concept acquisition or 
concept mastery depending upon the learning objectives. An entry 
test must be provided if the single group design is used. If the 
evaluator prefers to use a randomized , experimental-control design, 
he must find a new user group before moving ahead. Once the 
user group is selected, the evaluator muat determine if the user 
followed the directions precisely; any departures from the established 
procedures must be noted* 

The last criterial question concerns the results of using the 
protocol materials. These include the user's satisfaction with 
the materials as^well as the students the proportion of students 
v^ho had acquired the concept prior to the use of the protocol 

materials as determined by results of the entry test, the actual 

■J 

time spent by the students vefore they reached the criterion 
level, and the differences between the criterion performance of 
the treatment and control groups. 

If the protocol materials package attains the objectives 
and seems efficient at Evaluation Decision #9, the materials are 
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ready for summative evaluation. If the package was only partly 
successful, the director should be able, if he has satisfactorily 
accomplished each evaluation decision point, to ameliorate the 
situation. 

Although each projector director field tests his protocol 
materials as he develops them, the National Center for the Improvement 
of Educational Systems also has a grant with the Florida State 
Department of Education for field testing on a systematic basis 
all the materials developed in the protocol materials projects. 
The overall plan for field testing the materials provides data on 
the materials, collected at both the pre- and in-service levels. 

As the materials under development are completed, the Florida 
State Department project staff will list and describe each concept 
exemplified in the protocol materials produced by each project and 
discuss the development process used for each of the materials. 
After the field testing has been completed, a review board will 
examine the data from the field test and determine, on the basis 
of specified criteria, which of the protocol materials products 
are ready for widespread dissemination. 
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Appendix A 

Pilot Protocol Materials Projects, Funded FY 1971 



1. California ftate University, Northridge — Protocol Materials in 
English 

2. Stanford University-Far West Regional Laboratory — Protocol 
Materials in Teacher and Student Behavior, Teacher-Student 
Interaction in the eiassroom. 

3. University of Colorado — Protocol Materials in Instruction as an 
Interactive Process 

4. Florida State Department of Education — Field Testing of Materials 
Produced in Protocol Materials Projects 

5. Indiana University — Protocol Materials in Cognitive and Affective 
Interaction and Classroom Management 

6. Harvard University — Educational Development Corporation Protocol 
Materials Relationship Between Learning Behavior and Conceptual 
Demands of Subject 

7. Washington University — Protocol Materials in Teaching Concepts 
Teaching Interpreting, and Teaching Particulars 

8. Michigan State University — Protocol Materials in Classroom 
Interaction (Model Learning, Respondent Learning, Reinforcement^ 
Operant Learning, Shaping) 

9. Ohio State University — Protocol Materials in Oral Language 
Acquisition 

10. Teaching Research, Oregon State System of Higher Education, Protocol 
Materials in Learner Outcomes 

11. Bucknell University — Protocol Materials in Reading 
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FOREWORD 

"Progressive Evaluation of Protocol Materials Development" 
was originally prepared for use by protocol materials developers 
in a workshop at Michigan State University' in October^ 1971. 
The term '^progressive evaluation" is used to describe the step- 
wise or progressive nature of evaluation which occurs when 
development moves through successive phases. '^Protocol 
materials" are instructional materials, usually employing 
audio tape, video tape or film, intended to illuminate a concept 
by showing instances to which the concept correctly applies. 
These instances typically involve the behaviors of children 
and adults as they appear in the classroom or in other community 
settings in which teachers might be expected to interpret 
behavior for purposes of education. 

Because the workshop was intended to promote the critical 
examination and discussion of protocol development in small 
groups, the document presented here appeared as a linear 
program of questions which required the production of some 
form of evidence in response, thus facilitating group 



4. 

discussion. On subsequent pages, the linear format is 
maintained, but since the reader cannot discuss or critique 
the various points with the writer, notes are provided to 
illuminate those questions and requests for evidence which 
may not immediately be clear. To facilitate reading, each 
frame of the program appears on the left-hand page, and the notes 
pertinent to that frame appear on the page opposite. To 
perform the program realistically, the reader should get a 
concept significant to teaching ix^^mind, begin at Start , 
then respond to each question with the appropriate type of 
evidence. 



Notes for Frame 1 



5, 



The criteria- in-raind for a concept are those characteristics 
of things or processes which enable one to reliably identify 
them as belonging to a particular class. When the concept 
is considered ^'closed", the necessary and sufficient character- 
istics for a thing tc be placiBd in the class are known. Thus, 
the necessary and sufficient characteristics of "reinforcement" 
are: 1) that it follows a response 9 and 2) that it increases 
the probability of the response It follows* An "open" concept 
is one for which the salient characteristics may be identified, 
but the necessary and sufficient characteristics to exhaustively 
define the meaning of it cannot be stated. Concepts labeled 
by such terms as ^Vreativity", "ego", "anxiety", "democracy", 
may be viewed as open. Although the criteria-in-mind for 
closed concepts can usually be more easily stated than for 
open concepts, in both instances it it is very important 
for the protocol developer to be able to sketch instances 
to which the concept correctly applies. When many instances 
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Notes for Frame 2 
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In the social sciences many significant concepts cluster 
together within a theoretical system. Thus in Freudian psychology, 
anxiety and the attendant defense mechanisms are to be understood 
relative to other concepts such as id, superego and ego. In 
Skinner ian psychology, reinforcement , shapint , and extinction 
belong to a cluster, while Rogerian theory, self, threat, and 
anxiety cluster together. In social psychology task role, 
social role and leadership style cluster and must be carefull)'' 
distinguished from each other. If the protocol developer is 
careful to observe such clusters, families of protocol materials 
may be developed. 

A difficulty with many concepts in the social sciences 
is that they are wholly inferential entities or "constructs". 
A construct is not directly observable and cannot be 
instanced, although "indicators" of the construct may be 
instanced. Thus, ego processes cannot be directly instanced^ 



but ego defense mechanisms , which are taken to be indicators 
of the presence of ego processes in Freudian theory, can be 
instanced. 



Many social science concepts are of theoretical importance 
within a particular discipline, but may be of limited utility 
to a teacher in interpreting behavior and in subsequently 
taking some action with respect to that behavior. Thinking of 
situations in which a teacher might employ the concept one 
has in mind for a protocol provides a test of its practicality 
for teachers and helps one judge whether or not the concept 
is truly a significant one. 



4J o 

o 



CO 

I 4J 

0) 



o 
o 



X 



W 
CO 

O fH. 
00 O 



U 

o 

u 
o 
u 



0: Ch 



o o 

o ^ 



o 



4J 

O 
0) 



i 



I 

o « 
o 

4^ a 

•r-l 

u 

fx 

QJ QJ 

u ' 

Q> O 

w 



o 

•H 

u 



3 
P. 

o 

Ok 



A-i I 

^ W 

•H -H 

0) o 

o 

4J a* 



0 
o 

U 
'H 

•H 

cr 0 
o bO 
CO M 

O r4 











1 




















0 




0 




































CU 








> 


u 


P. 






w 


"J 






ij 














> 








QJ 




1 




4J 








w 


Si 




0 






w 


a> 




;3 


0 


0 


4J 










u 
















H 


Cu PC 














u 


<^ 








m 





o 

, CI 

a) o 

o 

w o 1-t 
H -U 



:3 
o 



4J 



t 

•H 

a, CI 



r 

H CD 
^ > 
o 



V3 

•H W W 



1 

o 

m 
o 



o 



<U (U <i) 

CO 4-> 

a QJ O 

4J Si 

(U O (U 

o O fS 

o o 



> 

•H C\J 



I 

o 
o 



1:3 

O 
•H 

JJ 

o 



H 



a 

u 

w <U C 



> . 

CO »H 

~ H 

M 

O CU 

o u 

CO o e 



o 

cu v< 



O XI 

cu 

C! U 

4J .H 



x> cu 
u 



zi a> <D 

O t2 -U 

d) to 

. cu W3 



0) 
CO 

23 



H erf 



o 

4J 



N 

00 

O 0 
0) 

cu cu 
o o 

4J O 



CM 



U 4J 

H TvJ 

cr -iJ 'o 



3 



cu 
o 

o cu o 

U C 

o o 



•H O 

cu 



ERIC 



I 

u 
o 

c 
1^ 



o 
u 

fS4 





0 XJ 


m 


u 


U 










1? 


U 




0 


CU 


0 


x: 




*H 




4J 


f/3 




cu 


•H iH J2 










0 














P 


cu 


•H 


0 




g 


•r4 


0 






cu 


CU 




w 


u 


ZJ 




0) 


rH 


M 










> 






W 


T-4 





t3 



CU 
O 

o 
o 



> 

t-Dx: 
p 

P a 
^ p 

a; rJ 



p cu 

CU O 
■r-i 
P ^ 

t-l p 



H nD 

^ P 0) 

p • . «^ 

M B 0 



CM 



11. 

Notes for Frame 3 
The difference between "concept acquisition" and "concept 
mastery" is viewed as one of degree. Concept acquisition implies 
that the learner can verbalize the characteristics which 
define the concept and/or recognize instances of the concept 
under controlled conditions and when a narrow range of possible 
instances are presented to him. Under these conditions, one 
may say that the learner has "acquired" the concept. Concept 
^j'astery implies that the learner can invariably recognize in- 
stances of the concept under a variety of simulated and real- 
life situations without special supporting cues or prompts. 

In most instances proizocol developers cannot provide appropriate 
conditions for completely testing concept master}*. What they 
can provide are cross-validity materials drawn from the 
total film or tape footage used to produce the protocol 
training materials. If one shoots two hours of film to develop 
a ten minute protocol on "probing" he should have, remaining 
in the footage, a large number of Instances of probing as well as 
numerous instances of questions which might seem like probing 
but are actually negative instances. Editing these positive 
and negative instances into test films accompanied by appropriate 
instructions and response forms provides a means by whici> to 
evaluate the degree to which the learner has progressed toward 
mastery . 
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Notes for Frame 4 
A critical consideration ±t> the design of protocol materials 
lies in their adaptability to different user needs. Some In- 
structors may wish to use them in large groups, some in small, and 
others in individual study carrels or by television. Some 
instructors will consider the concept highly significant in his or 
her instruction, others will consider the concept minor and wish 
to devote little time to it. Although each developer must develop 
his own strategy for meeting diverse user needs, a good strategy 
is to assume that the materials will be subjected to the most 
stringent demands — those in which the student is required to acquire 
or master the concept by individual study of self-administered 
materials. This . strategy is a good one because it permits the 
instructor to retreat toward less structured approaches such as small 
group discussion if he wishes to do so. The opposite approach, 
providing little structure for the use of the materials, places 
the instructur in the position of having to develop his own 
materials if he wishes the student to acquire the concept by 
self instruction, and, by increasing the time investment of the 
instructor, decreases the probability of use. 

For self-instruct-^' Aal use, massive quantities of written material 
are rarely functional. "Self -administered" may mean only that the 
student has contact with introductory material, can view or 
listen to the protocol, and can test himself on the testing filxos or 
^^ipes until adequate proficiency is achieved. 
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16, 

Notes on Frame 5 
The point of evaluation is to influence coafidence in the 
value of some thing or process. Generally, confidence in value is 
increased as the number of empirical tests for value increases. 
Several types of empirical tests may be recognized with respect 
to protocol materials, 

1, If the instructors and students report that the materials 
were satisfying and worthwhile, given the needs they wished to meet 
by using them, increased confidence in value occurs, 

2, If students moved from a low level of mastery of the 
concept (say 20%) prior to using the materials to a high level of 
mastery (say 80%) frllowing their use, confidence in value 

is increased, even though no comparative data (e,g,, from a control 
group) are available. If most of the students had 80% mastery 
to begin with, confidence in the value of the materials is 
decreased , since they teach a concept students already know 
and are therefore redundant, 

3, If randomly assigned students who are instructed by means 
the materials significantly out-perform a randomly assigned control 
instructed by other methods for equal time, confidence is increased. 
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Notes on Vmme 5 
The point of evaluation is to influence confidence in the 
value of some thing or process. Generally, confidence in value is 
Increased as the niunber of empirical tests for value increases • 
Several types of timpirical tests may be recognized with respect 
to protocol materials. 

1. If the instructors and students report that the materials 
were satisfying and worthwhile, given the needs they wished to meet 
by using them, increased confidence in value occurs. 

2. - If s :udents moved from a low level of mastery of the 
concept (say 20%) prior to using the materials to a high level of 
mastery (say 80%) following their use , confidence in value 

is increased, even though no comparative data (e.g., from a control 
group) are available. If most of the students had 80% mastery 
to begin with, confidence In the value of the materials is 
decreased , since they teach a concept students already know 
and are therefore redundant. 

3. If randomly assigned students who are instructed by means 
the materials significantly out-perform a randomly assigned control 
instructed by other methods for equal time, confidence is increased. 
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17. 

Moreover, as the number of alternative methods of instruction 
to which the protocols are superior increases, confidence in the 
value of the protocols correspondingly increases. If the protocol 
materials are superior to other methods for teaching the same 
concept(s), but require greater instructional time, confidence 
in value is not increased. 

4, As the number of different users who report satisf miction , 
increases in mastery, and comparative superiority for protocol 
materials increases, confidence in their value correspondingly 
Increases* 

A major point of failure in many evaluation efforts lies 
in ikdequate procedures or in the inadequate reporting of procedures; 
equal care in the evaluation procedures will help insure a product 
in which full confidence can be invested. 
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